Safe Policy Iteration

CONTRIBUTIONS 1. Theoretical contribution. We introduce a new, more general lower bound to the policy improvement of an arbitrary policy compared to another policy based on the ability to bound the distance between the future state distributions. 2. Algorithmic contribution. We define two approximate policy–iteration algorithms whose policy improvement moves toward the estimated greedy policy by maximizing the policy improvement bounds. 3. Empirical contribution. We report results on a simple chain walk and BlackJack domains that confirm the main theoretical findings. PROBLEM • Classical API approaches may generate a policy πt+1 that performs worst than the previous policy πt. • This undesired improvement may lead to the policy oscillation phenomenon that can prevent convergence to the optimal policy and degrade the learning process. • Our “safe” approach tries to overcome this issue visiting a sequence of policies with monotonic improving performance. Following this approach, the policy is constrained to improve overtime and, as a consequence, the degradation of the policy performance between consecutive iteration is prevented.